Apparatus, system, and method for coordinating storage requests in a multi-processor/multi-thread environment

ABSTRACT

An apparatus, system, and method are disclosed for coordinating storage requests in a multi-processor/multi-thread environment. An append/invalidate module generates a first append data storage command from a first storage request and a second append data storage command from a second storage request. The storage requests overwrite existing data with first and second data including where the first and second data have at least a portion of overlapping data. The second storage request is received after the first storage request. The append/invalidate module updates an index by marking data being overwritten as invalid. A restructure module updates the index based on the first data and updates the index based on the second data. The updated index is organized to indicate that the second data is more current than the first data regardless of processing order. The modules prevent access to the index until the modules have completed updating the index.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 12/098,435 entitled “Apparatus, System, and Methodfor Coordinating Storage Requests in a Multi-Processor/Multi-ThreadEnvironment” filed on Apr. 6, 2008 for David Flynn, et al., which is acontinuation-in-part of and claims priority to U.S. patent applicationSer. No. 11/952,121 entitled “Apparatus, System, and Method for a DeviceShared between Multiple Independent Hosts” and filed on Dec. 6, 2007 forDavid Flynn, et al., both of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to parallel and more particularly relates tocoordinating storage requests to a data storage device from multipleprocessors or threads.

2. Description of the Related Art

Traditional data storage devices, such as disk drives, optical drives,etc. often operate as a random access device to store data. Randomaccess is typically the ability to access an arbitrary element of asequence in equal time. This differs from sequential access where datais stored sequentially and also accessed sequentially such that accesstime varies significantly based on location of data. For example, a tapefor storing data is a sequential device. Accessing one data element cantake a different amount of time since one data element may be locatedclose to the where a read head is located and another data element maytake longer to access because the tape must be forwarded or reversed toget to the second data element.

A random access device, such as a memory chip, a disk drive, orsolid-state storage, operates as a random access device because accesstime to every data element on the random access device requiresapproximately the same access time. Random access is desirable becauseof the predictable nature and efficiency of data access. Some devices,such as memory chips, support both random read and random write,typically supporting the ability to overwrite data with updated data.Some devices, such as flash memory, support random read, but require anentire region of memory to be erased (e.g., an erase block or eraseregion) before data can be written into the region.

Typically, a file system communicates with a random access storagedevice using low level commands. The file system manages where the datais placed. The low level commands usually include a physical address anda data length in a command to store or access data. Data in a randomaccess storage device may be updated in a read-modify-write operationwhere data at a particular address is read, modified, and then writtento the same location where it was stored.

Data storage devices are often accessed by multiple devices. Usually afile server manages access to the data storage device and coordinatesrequests that involve common data. For a typical random access datastorage device, once a storage request is submitted by a file server,the file server typically must wait until the request is complete beforesending another storage request to ensure that data is written in theproper order. Where a data storage device takes a relatively long timeto complete a particular storage request, processing times can berelatively lengthy.

Some data storage devices store data sequentially while data accesstimes for the data storage devices are approximately equal. For example,data is stored sequentially in some solid-state storage devices, such asflash memory. Data access times for solid-state storage devices arefairly uniform, like other random access devices. Using a traditionalfile server or other data management system to coordinate requests fromdifferent devices for a sequential storage device impacts performance.For example, during a read-modify-write operation, modified data is notwritten back to the same location, but is instead sequentially stored.Where a file server manages the update, the file server must wait forthe entire update process to complete before servicing another datarequest to the same file or object. When metadata such as that stored indirectories and maps is locked while one append process completes anupdate, other processes or threads may be locked out from accessing themetadata. This update includes waiting for the sequential storage deviceto return a physical location where the data was stored sequentially.

In a parallel computing environment, waiting for a data storage deviceto complete a storage request is inefficient because parallel processorsmay be held up waiting for a single thread storage process to complete.For a sequential storage system implementing a low-level, log-basedstorage process, what is needed is an optimistic completion approachthat allows many threads or processes to execute in parallel.

SUMMARY OF THE INVENTION

From the foregoing discussion, it should be apparent that a need existsfor an apparatus, system, and method that allows a data storage deviceto interleave processing of storage requests so that multiple processorsor multiple threads can complete various portions of storage requests inparallel, even when the storage requests relate to a common file orobject. Beneficially, such an apparatus, system, and method wouldincrease performance in a multi-processor, multi-thread environmentwhile ensuring that modified data reflects a proper update order basedon order of arrival of the storage requests.

The present invention has been developed in response to the presentstate of the art, and in particular, in response to the problems andneeds in the art that have not yet been fully solved by currentlyavailable parallel processing systems. Accordingly, the presentinvention has been developed to provide an apparatus, system, and methodfor coordinating storage requests between multiple processors or threadsthat overcome many or all of the above-discussed shortcomings in theart.

The apparatus to coordinate storage requests is provided with aplurality of modules configured to functionally execute the necessarysteps of coordinating servicing of two storage requests that access asingle data segment. These modules in the described embodiments includean append/invalidate module and a restructure module.

The append/invalidate module generates a first append data storagecommand in response to receiving a first storage request. The firststorage request includes a request to overwrite existing data with firstdata. The existing data is part of a data segment stored on a datastorage device. The append/invalidate module generates a second appenddata storage command in response to receiving a second storage request.The second storage request includes a request to overwrite existing dataof the data segment with second data. The first and second data includeat least a portion of overlapping data to be stored at a common offsetwithin the data segment. The second storage request is received afterthe first storage request. The append/invalidate module updates an indexin response to the first storage request by marking data of the datasegment as invalid. The data marked invalid includes data being replacedby the first data. The append/invalidate module updates the index inresponse to the second storage request by marking data of the datasegment as invalid. The data marked invalid includes data being replacedby the second data.

The restructure module updates the index based on the first data andupdates the index based on the second data. The updated index isorganized to indicate that the second data is more current than thefirst data when either the index is updated based on the first databefore being updated based on the second data or the index is updatedbased on the second data before being updated based on the first data.The append/invalidate module prevents access to the index until theappend/invalidate module has completed updating the index and therestructure module prevents access to the index until the restructuremodule has completed updating the index.

In one embodiment, the apparatus includes a data location update modulethat updates the index with a first physical location indicating wherethe data storage device stored the first data. The data location modulealso updates the index with a second physical location indicating wherethe data storage device stored the second data. The physical locationswhere the data storage device stored the first data and the second dataare received from the data storage device. The updated index isorganized to indicate that the second physical location is more currentthan the first physical location when either the index is updated basedon the first physical location before being updated based on the secondphysical location or the index is updated based on the second physicallocation before being updated based on the first physical location. Thedata location update module prevents access to the index until the datalocation module has completed updating the index.

In another embodiment, processing a storage request by theappend/invalidate module occurs before processing by the restructuremodule and processing of the storage request by the restructure moduleoccurs before processing by the data location module. In anotherembodiment, processing of the first storage request by theappend/invalidate module, the restructure module, or the data locationupdate module occurs prior to or after processing of the second storagerequest by the append/invalidate module, the restructure module, or thedata location update module.

In one embodiment, an instance of each of the append/invalidate module,the restructure module, and the data location update module are executedin two or more threads and each instance of the append/invalidatemodule, the restructure module, and the data location update moduleprevents access to the index until the append/invalidate module, therestructure module, or the data location update module has completedupdating the index. In another embodiment, an instance of each of theappend/invalidate module, the restructure module, and the data locationupdate module are executed on two or more processors operating inparallel and each instance of the append/invalidate module, therestructure module, and the data location update module prevents accessto the index until the append/invalidate module, the restructure module,or the data location update module has completed updating the index.

In one embodiment, the apparatus includes a read module that reads atleast a portion of the data segment in response to a storage requestincluding a read request. In the embodiment, the apparatus includes aread delay module that delays servicing the requested read until thefirst storage request is serviced by the append/invalidate module, therestructure module, and the data location update module. In a furtherembodiment, the storage request to read at least a portion of the datasegment is received by the apparatus after the first storage request andbefore the second storage request and the read delay module delaysservicing the read request until both the first and second storagerequests are serviced by the append/invalidate module, the restructuremodule, and the data location update module.

In one embodiment, the apparatus includes a sequence number module thatassociates a sequence number with a storage request. Assigned sequencenumbers represent an order that storage requests are received by theapparatus. The restructure module organizes the index to indicate thatthe second data is more current than the first data by using a sequencenumber assigned to each of the first storage request and the secondstorage request. In another embodiment, organizing the index to indicatethat the second data is more current than the first data includesupdating a portion of the index mapped to the overlapping data with thesecond data and retaining the mapping to the second data if therestructure module updates the index based on the second data beforeupdating the index based on the first data.

In another embodiment, the append/invalidate module receives three ormore storage requests to overwrite existing data of the data segment andthe restructure module organizes the index with data of the storagerequests based on an order of arrival of the storage requests regardlessof an order that the restructure module uses data of the storagerequests to update the index. In another embodiment, the index isfurther divided into two or more regions and a storage requestpertaining to a first region of the index is serviced while additionalinstances of the one or more of the append/invalidate module and therestructure module prevent access to a second region of the index whileservicing a storage request pertaining to a second region. In anotherembodiment, the first storage request is transmitted from a first clientand the second storage request is transmitted from a second client.

In one embodiment, updating the index in response to the first storagerequest by marking the data of the data segment being replaced asinvalid also includes marking one or more bits of a bitmap. The bits inthe bitmap correspond to one or more locations in the data storagedevice where the data of the data segment being replaced by the firstdata and the second data resides. In another embodiment, updating theindex in response to the first storage request by marking the data ofthe data segment being replaced as invalid also includes temporarilymarking the entire data segment invalid until the restructure moduleupdates the index based on the first data and the second data and thenupdating the index to indicate that data of the data segment other thanthe data being replaced is valid. In yet another embodiment, updatingthe index in response to the first storage request by marking the dataof the data segment being replaced as invalid also includes generating alist indicating portions of the data segment being replaced by the firstdata and the second data are invalid and indicating that portions of thedata segment not being replaced by the first data and the second dataare valid.

In one embodiment, the first storage request and/or the second storagerequest are received substantially without data. In another embodiment,the first append data storage command and/or the second append datastorage command are transmitted to the data storage device substantiallywithout data. In another embodiment, at least one of a storage requestand an append data storage command initiates a direct memory access(“DMA”) process or a remote DMA (“RDMA”) process to transfer data of thedata segment to the data storage device.

A system of the present invention is also presented to coordinatestorage requests. The system may be embodied by a data storage deviceand a storage controller that controls the data storage device. Inparticular, the storage controller, in one embodiment, includes anappend/invalidate module and a restructure module.

The append/invalidate module generates a first append data storagecommand in response to receiving a first storage request. The firststorage request includes a request to overwrite existing data with firstdata. The existing data is part of a data segment stored on the datastorage device. The append/invalidate module generates a second appenddata storage command in response to receiving a second storage request.The second storage request includes a request to overwrite existing dataof the data segment with second data. The first and second data includeat least a portion of overlapping data to be stored at a common offsetwithin the data segment. The second storage request is received afterthe first storage request. The append/invalidate module updates an indexin response to the first storage request by marking data of the datasegment as invalid. The data marked invalid includes data being replacedby the first data. The append/invalidate module updates the index inresponse to the second storage request by marking data of the datasegment as invalid. The data marked invalid includes data being replacedby the second data

The restructure module updates the index based on the first data andupdates the index based on the second data. The updated index isorganized to indicate that the second data is more current than thefirst data when either the index is updated based on the first databefore being updated based on the second data or the index is updatedbased on the second data before being updated based on the first data.The append/invalidate module prevents access to the index until theappend/invalidate module has completed updating the index and therestructure module prevents access to the index until the restructuremodule has completed updating the index.

In one embodiment, the data storage device is a solid-state storagedevice. In another embodiment, first storage request and the secondstorage request are received from separate requesting devices where arequesting device may be a client or a server. In another embodiment, atleast one of the requesting devices is a client and theappend/invalidate module and the restructure module operate independentof the client. In another embodiment, the system includes a server andthe storage controller is within the server.

A method of the present invention is also presented for coordinatingstorage requests. The method in the disclosed embodiments substantiallyincludes the steps necessary to carry out the functions presented abovewith respect to the operation of the described apparatus and system. Inone embodiment, the method includes generating a first append datastorage command in response to receiving a first storage request tooverwrite existing data of a data segment stored on a data storagedevice with first data. The method includes generating a second appenddata storage command in response to receiving a second storage requestto overwrite existing data of the data segment with second data. Thefirst and second data include at least a portion of overlapping data tobe stored at a common offset within the data segment. The second storagerequest is received after the first storage request.

The method includes updating an index in response to the first storagerequest by marking data of the data segment as invalid where the datamarked invalid includes data being replaced by the first data. Themethod includes updating the index in response to the second storagerequest by marking data of the data segment as invalid, where the datamarked invalid comprising data being replaced by the second data. In oneembodiment, access to the index is prevented until an append datastorage command is generated for the first storage request or the secondstorage request and the index is updated by marking the data of the datasegment invalid for one of the first storage request and the secondstorage request.

The method includes updating the index based on the first data andupdating the index based on the second data. The updated index isorganized to indicate that the second data is more current than thefirst data when either the index is updated based on the first databefore being updated based on the second data or the index is updatedbased on the second data before being updated based on the first data.Access to the index is prevented until the index is updated based on oneof the first data and the second data.

Reference throughout this specification to features, advantages, orsimilar language does not imply that all of the features and advantagesthat may be realized with the present invention should be or are in anysingle embodiment of the invention. Rather, language referring to thefeatures and advantages is understood to mean that a specific feature,advantage, or characteristic described in connection with an embodimentis included in at least one embodiment of the present invention. Thus,discussion of the features and advantages, and similar language,throughout this specification may, but do not necessarily, refer to thesame embodiment.

Furthermore, the described features, advantages, and characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. One skilled in the relevant art will recognize that theinvention may be practiced without one or more of the specific featuresor advantages of a particular embodiment. In other instances, additionalfeatures and advantages may be recognized in certain embodiments thatmay not be present in all embodiments of the invention.

These features and advantages of the present invention will become morefully apparent from the following description and appended claims, ormay be learned by the practice of the invention as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem for converting a storage request to an append data command inaccordance with the present invention;

FIG. 2 is a schematic block diagram illustrating one embodiment of anapparatus for converting a storage request to an append data command inaccordance with the present invention;

FIG. 3 is a schematic block diagram illustrating one embodiment of analternate apparatus for converting a storage request to an append datacommand in accordance with the present invention;

FIG. 4 is a schematic flow chart diagram illustrating one embodiment ofa method for converting a storage request to an append data command inaccordance with the present invention;

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofanother method for converting a storage request to an append datacommand in accordance with the present invention;

FIG. 6 is a schematic block diagram of an example of converting astorage request to an append data command in accordance with the presentinvention;

FIG. 7 is a schematic block diagram illustrating one embodiment of anapparatus for efficient mapping of virtual and physical addresses inaccordance with the present invention;

FIG. 8 is a schematic block diagram illustrating another embodiment ofan apparatus for efficient mapping of virtual and physical addresses inaccordance with the present invention;

FIG. 9 is a schematic flow chart diagram illustrating one embodiment ofa method for efficient mapping of virtual and physical addresses inaccordance with the present invention;

FIG. 10 is a schematic flow chart diagram illustrating anotherembodiment of a method for efficient mapping of virtual and physicaladdresses in accordance with the present invention;

FIG. 11 is a schematic block diagram of an example of a forward map anda reverse map in accordance with the present invention;

FIG. 12 is a schematic block diagram illustrating one embodiment of anapparatus for coordinating storage requests in accordance with thepresent invention;

FIG. 13 is a schematic block diagram illustrating another embodiment ofan apparatus for coordinating storage requests in accordance with thepresent invention;

FIG. 14 is a schematic flow chart diagram illustrating one embodiment ofa method for coordinating storage requests in accordance with thepresent invention;

FIG. 15 is a schematic flow chart diagram illustrating anotherembodiment of a method for coordinating storage requests in accordancewith the present invention;

FIG. 16A is a first part of a schematic block diagram illustrating anexample of an apparatus for coordinating storage requests in accordancewith the present invention;

FIG. 16B is a second part of a schematic block diagram illustrating anexample of an apparatus for coordinating storage requests in accordancewith the present invention; and

FIG. 16C is a third part of a schematic block diagram illustrating anexample of an apparatus for coordinating storage requests in accordancewith the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

The schematic flow chart diagrams included herein are generally setforth as logical flow chart diagrams. As such, the depicted order andlabeled steps are indicative of one embodiment of the presented method.Other steps and methods may be conceived that are equivalent infunction, logic, or effect to one or more steps, or portions thereof, ofthe illustrated method. Additionally, the format and symbols employedare provided to explain the logical steps of the method and areunderstood not to limit the scope of the method. Although various arrowtypes and line types may be employed in the flow chart diagrams, theyare understood not to limit the scope of the corresponding method.Indeed, some arrows or other connectors may be used to indicate only thelogical flow of the method. For instance, an arrow may indicate awaiting or monitoring period of unspecified duration between enumeratedsteps of the depicted method. Additionally, the order in which aparticular method occurs may or may not strictly adhere to the order ofthe corresponding steps shown.

FIG. 1 is a schematic block diagram illustrating one embodiment of asystem 100 for converting a storage request to an append data commandand to efficiently map physical and virtual addresses in accordance withthe present invention. The system 100 includes a storage device 102 thatincludes a storage controller 104 and a data storage device 106. Thestorage device 102 is within a server 108 connected to one or moreclients 110 through a computer network 112.

In one embodiment, the system 100 includes a storage device 102 with astorage controller 104 and a data storage device 106. The storagecontroller 104 and data storage device 106 may be included in a singleenclosure that is the storage device 102. In another embodiment, thestorage controller 104 and the data storage device 106 are separate. Thestorage controller 104 typically controls data storage and access forthe data storage device 106. The data storage device 106, in oneembodiment, is capable of substantially similar access times to datathroughout the data storage device 106. For example, the data storagedevice 106 may be a solid-state storage device, such as flash memory,nano random access memory (“nano RAM or NRAM”), magneto-resistive RAM(“MRAM”), dynamic RAM (“DRAM”), phase change RAM (“PRAM”), etc. The datastorage device 106 may also be a hard disk drive, a compact disk (“CD”)drive, an optical drive, and the like.

While the data storage device 106 is depicted in FIG. 1 as a singlestorage device, the data storage device 106 may include two or morestorage devices. The data storage devices 106 may be configured as aredundant array of independent drives (“RAID”), just a bunch of disks(“JBOD”), and the like. The data storage devices 106 may be configuredwith one or more data storage devices 106, such as solid-state storage,configured as high-performance, short-term storage and one or more datastorage devices 106, such as hard disk drives, configured aslower-performance, long-term storage. In this embodiment, the storagecontroller 104 may manage the various types of data storage devices 106.One of skill in the art will appreciate other types and configurationsof data storage devices 106.

The storage controller 104 may control one or more data storage devices106 and may be a RAID controller, a controller for a storage areanetwork (“SAN”), etc. The storage controller 104 may include one or moresub-controllers. The storage controller 104 may be integrated with thedata storage device 106 or separate and may be integrated together ordistributed. For example, a portion of the storage controller 104 may bea master controller and other portions of the storage controller 104 maybe sub-controllers or slave controllers. The master controller may be adevice in communication with other sub-controllers that in turn controldata storage devices 106, or may be a master controller that controlsslave controllers as well as a data storage device 106. One of skill inthe art will recognize other forms and functions of a storage controller104.

In one embodiment, the storage device 102 is included in a server 108.In various embodiments, either the storage controller 104 or datastorage device 106 or both may be located external to the server 108.The server 108 may be connected to the storage controller 104 or thestorage controller 104 may be connected to the data storage device 106over a system bus, such as, such as a peripheral component interconnectexpress (“PCI-e”) bus, a Serial Advanced Technology Attachment (“serialATA”) bus, or the like. In another embodiment, the solid-state storagedevice 102 is external to the server 108 or storage device 102 and maybe connected through a universal serial bus (“USB”) connection, anInstitute of Electrical and Electronics Engineers (“IEEE”) 1394 bus(“FireWire”), etc. In other embodiments, the storage device 102 isconnected to the server 108 or the storage controller 104 is connectedto the data storage device 106 using a peripheral component interconnect(“PCI”) express bus using an external electrical or optical busextension or networking solution such as Ethernet, Fibre Channel,Infiniband, or PCI Express Advanced Switching (“PCIe-AS”), or the like.One of skill in the art will recognize a wide variety of possibleconnection methods.

The server 108 may also instead be a personal computer, lap-topcomputer, mainframe computer, workstation, electronic device, etc. Theserver 108 may include a client 110 or be connected to a client 110 overa computer network 112. The system 100 may include any number ofcomputers, clients 110, computer networks 112, or other electronicdevice, as long as the system 100 is capable of transmitting a storagerequest to the storage device 102. The client 110 may be a processrunning on the server 108 or on another computer or electronic device.The client 110 may also be a personal computer, lap-top computer,mainframe computer, workstation, electronic device, etc. One of skill inthe art will recognize other components and configurations of a system100 capable of transmitting a storage request to the storage device 102.

FIG. 2 is a schematic block diagram illustrating one embodiment of anapparatus 200 for converting a storage request to an append data commandin accordance with the present invention. The apparatus 200 includes astorage request receiver module 202, a translation module 204, and amapping module 206, which are described below. The apparatus 200 isdepicted in FIG. 2 as being in the storage controller 104, but all or aportion of the modules 202, 204, 206 may be located external to thestorage controller 104 and may be distributed through various componentsof the system 100.

The apparatus 200 includes a storage request receiver module 202 thatreceives a storage request from a requesting device. In one embodiment,the requesting device is the server 108. In another embodiment, therequesting device is a client 110. The requesting device may be anydevice capable of sending a storage request.

The storage request includes a request to store a data segment of a fileor of an object onto the data storage device 106. The storage requestmay be an object request, a file request, a logical block storagerequest, and the like. The storage request includes one or more sourceparameters for the data segment. The source parameters include a virtualaddress of a file or object from which the data of the data segment wasderived. Typically a virtual address is an identifier for a file orobject. The virtual address may be file name, an object name, or otheridentifier known to a file system connected to the storage device 102.

A distinction is made between a logical address and logical addressspace and a virtual address and a virtual address space. In the presentcontext, a virtual address space is intended to encompass the broadestpossible range of indirect addressing. As used herein, a virtual addressspace may simultaneously comprise: one or more virtual address spaces,one or more logical address spaces, one or more memory spaces, one ormore logical block address spaces, one or more fixed block addressspaces, etc.

For example, a client may be operating multiple virtual operatingsystems. In this embodiment, each virtual operating system may have avirtual memory space. Each of these virtual memory spaces may be mappedto data stored within the storage device according to the device'svirtual address space and virtual-to-physical mappings. In anotherexample, objects within several clients may be independently mapped tothe same stored data in a many-to-one relationship, referencing theshared data with virtual addresses unique to the client. While examplesshown herein are one-to-one relationships, the methods, apparatuses, andsystems are intended to support many-to-one, one-to-many, and evenmany-to-many virtual-to-physical address mappings.

It is intended that the virtual-to-physical mapping methods supportsparse addressing (over-provisioning of the physical address space),thin provisioning, virtual partitions, and data transformations (e.g.compression, encryption) by supporting broad indirection inidentification, address, length, and metadata transformation.

As a convention herein, a virtual ID uniquely identifies the stored dataentity within the virtual space of the client. A virtual address morespecifically addresses the data for the virtual entity. For example, avirtual address may comprise a virtual ID and an offset within thedataset. In another example, a virtual address may comprise a virtual IDand an index within the virtual entity, where the index may be to arecord within a structure of non-uniform (e.g. variable length) records.

In one embodiment, the apparatus 200 emulates a logical block storagedevice and the source parameters include one or more logical blockaddresses where the data segment is requested to be stored by therequesting device through storage request. In this embodiment, thevirtual address may comprise the logical block addresses. For example,if the storage request is a logical block storage request, therequesting device may dictate where the data segment is intended to bestored in the data storage device 106. The logical block address mayinclude information such as a RAID group number, data storage deviceidentifier, partition, cylinder number, sector, offset, etc. One ofskill in the art will recognize other elements that may be included in alogical block address.

The storage request may include a request to store the data segment inmore than one location or may include a request to store portions of thedata segment in more than one location, so the storage request mayinclude more than one logical block address. Where a storage requestincludes a logical block address, the storage request typically alsoincludes one or more offsets and data lengths corresponding to the oneor more logical block addresses. An offset and data length may beimplicit if the logical blocks are of fixed size. An offset is typicallyhow far into a file or object, typically from a beginning of the file orobject, a data segment begins. The data lengths typically include howmuch of a storage device will be occupied by the data segment or aportion of the data segment associated with a logical block address.Typically, the offset and data length will be expressed using some unitrecognizable to the storage controller 104 and data storage device 106.For example, an offset and a data length may be expressed in terms ofbytes, blocks, sectors, or other unit used to divide the data storagedevice 106. One of skill in the art will recognize other ways to expressan offset and a data length for all or a portion of a data segment.

The system 100 includes a translation module 204 that translates thestorage request to one or more storage commands where at least onestorage command is an append data storage command. Each append datastorage command directs the data storage device 106 to store datacreated from the data segment and one or more source parameters at oneor more append points. The source parameters are stored with the dataand at least one of the source parameters is a virtual address.

The data storage device 106 preferably stores data as a data packet. Adata packet includes data of the data segment along with a data packetheader. In one embodiment, the data packet header includes sourceparameters. In another embodiment, the source parameters are storedsimultaneously with the data. For example, the data may be storedsequentially at an append point in one location on the data storagedevice while the source parameters are stored simultaneously andsequentially at another location in the data storage device. In theembodiment, the sequence in which the source parameters and data arestored may be used to pair the data and source parameters during a readoperation or other operation where the data and source parameters areretrieved.

In one embodiment, the data storage device 106 stores data (or datapackets) sequentially by storing data in a page, division, or otherspecified region, moving an append point to the next available addressjust past the end of the previously stored data, storing data at theappend point, again moving the append point to the next availableaddress past the data just stored, etc. Data is stored in a page,division, etc. until the page or division is full, then the append pointis moved and the data is stored in another page, division, etc. Appendpoints may be moved independently by the storage device 102, and inresponse to specific requests.

Sequentially storing data is particularly beneficial for solid-statestorage devices because it allows even distribution of data to preventhot spots or addresses that are written to more frequently than otheraddresses. Sequentially storing data is particularly beneficial forsolid-state storage devices as it eliminates seek times, eliminatesread-modify-write operations with related erasures and thus increasesdata reliability and the useful life of the solid-state storage device.Sequential storage in a solid-state storage device also typically doesnot adversely impact read access time because a typical solid-statestorage device has about the same read access time for data storedanywhere in the solid-state storage device. This feature allows the datastorage device 106 to emulate a random access device to effectivelyeliminate latencies due to write seek times and increase the mediareliability and useful life of the solid-state storage device 106,without negatively impacting read performance.

Sequential storage may have other benefits as well for the data storagedevice 106. The benefits of sequential storage as related to access timeare more fully described in the U.S. patent application Ser. No.11/952,095 entitled “Apparatus, System, and Method for Managing Commandsof Solid-State Storage Using Bank Interleave” and U.S. patentapplication Ser. No. 11/952,101 entitled “Apparatus, System, and Methodfor Storage Space Recovery In Solid-State Storage” [hereinafter “StorageSpace Recovery Application”], both for David Flynn et al. and filed Dec.6, 2007, both herein incorporated by reference.

One significant benefit is that by storing data sequentially and bystoring the source parameters with the data (in a packet header orsimultaneously) the data storage device 106 a log storage device. A logstorage device typically keeps track of a sequence or order of datastorage so that if a first data packet is stored after a second datapacket, this order of storage is known and can be determined.

In one embodiment, an append point where data is to be stored isindependent of context. Whereas data is stored sequentially, an appendpoint may be maintained by the storage device so that data received inconjunction with a storage request may be stored at the next availablephysical location within the data storage log. There is no externalcontext to this append point. Meaning that data is not stored indifferent physical regions of the device according to a explicit orimplicit relationship with the client. For example, a first client mayaccess the device using a first partition while a second client accessthe device using a second partition. These partitions are strictlylogical constructs within the virtual addressing scheme. The data forthe two clients, in the two disparate partitions, is still appendedsequentially. In this way, the device does not limit the number of openfiles, or thereby the number of clients that can access the devicesimultaneously. An additional benefit is that the storage space is usedwith optimal efficiency and naturally supports storage methods toimprove capacity utilization such as thin provisioning.

An append point is typically set to an address just after previouslystored data or data packets, but in other embodiments, may be set at abeginning address of a page, erase block, division, etc., may be setjust after a block of addresses that are unusable, etc. In oneembodiment, the data storage device 106 maintains an address of theappend point and the translation module 204 merely creates an appenddata storage command directing the data storage device 106 to store dataat the append point. Once data is stored, the data storage device 106then reports the physical address where the data was stored to themapping module 206 or to another device or module. In anotherembodiment, the translation module 204 is aware of or maintains thephysical address in the data storage device 106 of the append point andcreates an append data storage command using the physical address of theappend point.

In one embodiment, an append point, or erase region pointer, indicatinga next erase region (or erase block) to be written to after the currenterase region is filled may be queued up in advance and pulled from thequeue by the data storage device 106 or the translation module 204. Inanother embodiment, the append point (erase region pointer) is movedfrom sub-region to sub-region according to a prescribed pattern. Theprescribed pattern may comprise the region sequence information.

The append data storage command is typically a command to store data atan append point. The data is created from a data segment and typicallythe data of a data segment spans the data segment. The translate module204 may create one or more append data storage commands. For example, ifa data segment is broken up and more than one portion of the data arestored in non-contiguous locations, more than one append data storagecommand may be required. In another embodiment, a single append datastorage command is capable of storing the data segment in multiple,non-contiguous locations.

The data of the data segment may come from various sources. In oneembodiment, the data is data from a file that is new and not previouslystored on the data storage device 106. In another embodiment, the dataof the data segment has been read from the data storage device 106 andhas been modified prior to storing the data again as data packets in thedata storage device 106. In another embodiment, the data of the datasegment is from another erase region (such as an erase block), page,division, etc. being recovered in a storage recovery (garbagecollection) operation. In the embodiment, the data may be valid datathat is moved from a selected erase region prior to taking action torecover the erase region for future data storage. In another embodiment,the data is index data or mapping data that is being stored to protectan index or map. One of skill in the art will recognize other data thatmay be in a data segment received by the storage request receiver module202.

In various embodiments, the translation module 204 creates othercommands relevant to the storage request. For example, the translationmodule 204 may create a set append point command, a read command, anerase command, a reset command, a move command, a sync command, a flushcommand, a read control register command, a modify control registercommand, a program page command, an erase command directed at an eraseblock, a transfer command list command, a request status command, or thelike. The other commands typically supplement the append data storagecommand to service the storage request. One of skill in the art willrecognize other relevant commands and a sequence of commands that may becreated by the translate module 204 to service a storage request.

In one embodiment, the storage request received by the storage requestreceiver module 202 is received substantially free of data. In thiscase, the storage request is a request to transfer data and essentiallydoes not include the data. In another embodiment, the append datastorage command is transmitted to the data storage device 106substantially without data. In this case, the append data storagecommand is a command to transfer data and essentially does not includethe data. In a further embodiment, the source parameters include one ormore physical memory addresses within a host or client 110 where thedata segment is read from as a result of the storage request. In theembodiment, the storage request or command created by the translatemodule 204, such as the append data storage command, initiates orrequests a direct memory access (“DMA”) or remote direct memory access(“RDMA”) process to transfer data of the data segment to the datastorage device 106. For example, a DMA process may be initiated by anappend data storage command to DMA data from a client 110 to a locationwithin the data storage device 106. One of skill in the art willappreciate other ways to initiate or request a DMA or RDMA process.

In a typical DMA or RDMA process, the data storage device 106 pulls datafrom memory of a host during a write operation and pushes data to thehost during a read operation. This is beneficial because the host doesnot need to know where the data will be stored on the data storagedevice 10. The host can merely tell the storage device 102 where data tobe written is to be pulled from or where the data is to be stored for aread.

The apparatus 200 includes a mapping module 206 that maps one or moresource parameters of the data segment to one or more locations in thedata storage device 106 where the data storage device 106 appended thedata of the data segment and the source parameters. The sourceparameters may include a virtual identifier associated with the datasegment, a device identifier, a partition identifier, lengths of one ormore data packets of the data segment, one or more memory locations in amemory of a host where the data segment is located prior to orsubsequent to the storage request, one or more lengths of data in theone or more memory locations, attributes of the data segment, metadataof the data segment, control parameters of the data segment, and thelike.

The mapping between the source parameters of the data segment to thephysical locations where data of the data segment was storedbeneficially allows the apparatus 200 to emulate a random access deviceusing a data storage device 106 where data is stored sequentially. Thisis beneficial because a storage device 102 or storage controller 104with the apparatus 200 can be connected as a random access device andcan receive object requests, file requests, and logical block storagerequests without differentiating between the requests. The apparatus 200treats data from the various requests equally—mapping a logical blockaddress received in a storage request in essentially the same way as avirtual address. In other words, a logical block address, data length,etc. received in a logical block storage request may become a virtualaddress to be mapped to a physical address of a location where data ofthe data request is stored on the data storage device 106.

FIG. 3 is a schematic block diagram illustrating one embodiment of analternate apparatus 300 for converting a storage request to an appenddata command in accordance with the present invention. The apparatus 300includes a storage request receiver module 202, a translation module204, and a mapping module 206, which are substantially similar to thosedescribe above in relation to the apparatus 200 of FIG. 2. The apparatus300 includes a storage response receiver module 302, a responsetransmission module 304, a compression module 306, an index rebuildmodule 308, a command reordering module 310, a request reordering module312, and a garbage collection module 314, which are described below. Theapparatus 300 is depicted in FIG. 3 as being in the storage controller104, but all or a portion of the modules 202, 204, 206, 302-314 may belocated external to the storage controller 104 and may be distributedthrough various components of the system 100. In addition the modules202-206, 302-314 of the apparatus 300 may operate independent of theclient 110.

In one embodiment, the apparatus 300 includes a storage responsereceiver module 302 that receives one or more storage command responsesfrom the data storage device 106. The storage command responses includeone or more locations where the data storage device 106 appended thedata of the data segment. In the embodiment, the locations where thedata storage device 106 stored the data may be unknown to the apparatus300 until the data storage device 106 responds and indicates thelocations where the data was appended. When the physical locations wherethe data storage device 106 appended the data is unknown until after thedata storage device 106 stores the data, the mapping module 206 receivesthe one or more locations where the data storage device 106 appended thedata of the data segment from the data storage device 106, typicallyfrom the one or more storage command responses. In another embodiment,as discussed above, the translation module 204 tracks or managesphysical addresses where the data of the data segment are stored and themapping module 206 may receive the physical addresses of the locationswhere the data was stored from the translation module 204.

In another embodiment, the apparatus 300 includes a responsetransmission module 304 that transmits a storage request response to therequesting device. The storage request response includes informationregarding execution of the storage request. For example, the storagerequest response may indicate successful execution of the storagerequest or other status information. In another embodiment, the storagerequest response includes information indicating where the data andassociated source parameters were stored. This embodiment may not bedesirable if the apparatus 300 is emulating a random access device. Inone embodiment, the response transmission module 304 sends the storagerequest response after the storage response receiver module 302 receivesthe storage command responses indicating that all of the data of thedata segment and associated source parameters were successfully storedon the data storage device 106. In another embodiment, the responsetransmission module 304 sends the storage request response independentof receiving a storage command response. One of skill in the art willappreciate other information sent in a storage request response andtiming of sending the response.

In one embodiment, the apparatus 300 includes a compression module 306that compresses data of an append data storage command to form the dataprior to storage on the data storage device 106. Typically, thecompression module 306 changes the data length of a portion data (or adata packet) of the data segment. This affects where data is stored andsubsequent append points. In this case, each append point may be unknownuntil after compression. Where compression is used, the data storagedevice 106 or some module down steam of the compression module 306typically tracks append points and physical location of data on the datastorage device 106 and waits until data is compressed to determine datalength of a portion of data (or a data packet) and a subsequent appendpoint. Once an append point is known and the data is compressed, alocation, which may be in the form of a physical address, along withdata length can be reported back to the mapping module 206. In oneembodiment, the compression module 306 stores compression informationwith the compressed data, typically in a data packet header for thedata. One of skill in the art will recognize other features andconsequences of compressing data prior to storage on the data storagedevice 106.

In another embodiment, the apparatus 300 includes an index rebuildmodule 308 that rebuilds the mapping created by the mapping module 206for the data segment using one or more of the source parameters and thephysical location on the data storage device 106 of the data of the datasegment. To improve speed of access, the index is typically stored infaster, volatile memory such as DRAM that is subject to loss due topower failures, system resets, etc. Storing the source parameters withthe data in a sequential storage device creates a non-volatile, datarecord within a sequential log that the index rebuild module 308 uses tore-create index mappings between source parameters and a physicaladdress and data length.

Source parameters may be stored in a header, in a specific locationwithin a data packet, or at the end of the data packet. Typically, thesource parameters are stored in a header for the data packet. The datalength is typically stored in a data header so that if an index ormapping is unavailable, data can be searched sequentially. In oneembodiment, the index rebuild module 308 tracks through a region of thedata storage device 106, such as a page or erase block, to rebuild anindex that includes the mapping for the data segment.

Beneficially, physical locations of the data stored on the data storagedevice 106 along with the source parameters stored with the datacomprise the primary virtual-to-physical map. The mapping created by themapping module 206 comprises a secondary virtual-to-physical map. Thesecondary virtual-to-physical map is typically stored in RAM so that ifpower is lost, a failure occurs, or for some other reason the mapcreated by the mapping module 206 becomes invalid, the primaryvirtual-to-physical map may be used to recreate the secondaryvirtual-to-physical map.

For example, the index rebuild module 308 looks at a data header at thestart of a page of data in the data storage device 106. The indexrebuild module 308 reads the physical address of the first data packetthen reads the source parameters, including data length, in the header.The index rebuild module 308 then maps the source parameters in the datapacket to the physical address and data length of the data packet. Theindex rebuild module 308 then uses the data length to move to the nextdata packet. The index rebuild module 308 then repeats the rebuildprocess tracking through all data packets in the page to build thesecondary virtual-to-physical map. In this way, the data storage device106 is a sequential log that can be used to rebuild an index containingmappings between physical addresses and source parameters, such as avirtual identifier, offset, logical block address, source physicallength, etc.

In one embodiment, the index is periodically checkpointed, or stored innon-volatile memory at a particular point in time or in a particularstate. An order of when each page was filled with data is maintainedin-band and the order is correlated with checkpoints. If an indexbecomes unavailable, the most recent index corresponding to the latestcheckpoint can be retrieved. The index may then be brought current byreplaying the log starting at a location where a data packet was savedjust after the checkpoint. The index rebuild module 308 may besynchronized by sequentially tracking through the data packets from thedata packet stored after the checkpoint to the latest stored data packetin order to update the index to a current status. Beneficially, theindex rebuild module 308 allows a checkpointed index to be restoredefficiently and quickly.

In one embodiment, the apparatus 300 includes a command reorderingmodule 310 that modifies a sequence that two or more outstanding appenddata storage commands are executed. The command reordering module 310 isbeneficial to sequence commands in a more efficient way. In oneembodiment, a read command might be postponed until a write command forthe same data segment completes. In another embodiment, a storage device102 that supports multiple channels may allow commands destined for afirst channel to be postponed while that channel is busy and allow othercommands to other channels to continue until such time as the firstchannel is available. In another embodiment, when the storage requestreceiver module 202 receives two or more storage requests, the apparatus300 includes a request reordering module 312 that reorders a sequencethat the storage requests are serviced. The command reordering module310 and the request reordering module 312 are beneficial to sequencecommands and requests in a more efficient way.

In another embodiment, the apparatus 300 includes a garbage collectionmodule 314 that moves valid data from a storage region on the datastorage device 106 identified for recovery and that erases invalid datafrom the storage region prior to returning the storage region to a poolof available space within the data storage device 106 for subsequentdata storage. In this embodiment, the mapping module 206 updates themapping of the source parameters of the valid data to one or morelocations in the data storage device 106 to the new location where thedata storage device 106 appended the valid data and associated sourceparameters. In one embodiment, moving valid data from a region selectedfor recovery may be treated in the same way as other storage requests.

FIG. 4 is a schematic flow chart diagram illustrating one embodiment ofa method 400 for converting a storage request to an append data commandin accordance with the present invention. The method 400 begins and thestorage request receiver module 202 receives 402 a storage request froma requesting device, such as a client 110 or a server 108. The storagerequest includes a request to store a data segment of a file or objectonto the data storage device 106. The storage request may include one ormore source parameters for the data segment that at least include one ormore logical block addresses where the data segment is requested to bestored by the storage request and one or more data lengths correspondingto the one or more logical block addresses.

The translation module 204 translates 404 the storage request to one ormore storage commands. At least one of the storage commands is an appenddata storage command. Each append data storage command directs the datastorage device 106 to store data of the data segment at one or moreappend points. An append point is a location in the data storage device106 that is a next address after the latest data segment that was storedon the data storage device 106. If the translation module 204 breaks thedata segment into more than one segment, typically more than one dataappend command is created. This may be required if data resulting fromthe data segment will not fit at the end of a page, erase block, etc. atthe append point. A second append point may be set at the beginning ofanother page, erase block, etc.

The mapping module 206 maps 406 one or more source parameters of thedata segment to one or more locations in the data storage device 106where the data storage device 106 appended the data of the data segmentand the method 400 ends. Typically the mapping is part of an index thatallows future access to the data. By mapping the physical locations ofthe data and data lengths to source parameters, the apparatus 200 canemulate a random access device while storing the data sequentially onthe data storage device 106.

FIG. 5 is a schematic flow chart diagram illustrating one embodiment ofanother method 500 for converting a storage request to an append datacommand in accordance with the present invention. The method 500 beginsand the storage request receiver module 202 receives 502 a storagerequest from a requesting device, such as a client 110 or a server 108.The translation module 204 translates 504 the storage request to one ormore storage commands where at least one of the storage commands is anappend data storage command. Again, each append data storage commanddirects the data storage device 106 to store data of the data segment atone or more append points.

The compression module 306 compresses 506 data of the one or more appenddata storage commands related to the data segments into compressed dataand the data storage device 106 stores 508 the compressed data inresponse to the append data storage commands. The storage responsereceiver module 302 receives 510 one or more storage command responsesfrom the data storage device 106. The storage command responses includeone or more locations where the data storage device 106 appended thedata of the data segment. Based on the storage locations received aspart of the storage command responses, the mapping module 206 maps 512one or more source parameters of the data segment to the one or morelocations in the data storage device 106 where the data storage device106 appended the data of the data segment and the method 500 ends.Compressing the data typically necessitates mapping the sourceparameters to the storage locations after the data is stored becausecompression typically changes the data length of the data.

FIG. 6 is a schematic block diagram of an example 600 of converting astorage request to an append data command in accordance with the presentinvention. The example 600 is merely an illustration of one embodimentof an apparatus 200, 300 for converting a storage request to an appenddata storage command and is not intended to be limiting in any way. Oneof skill in the art will recognize that there are many ways to implementthe present invention that are different than the example 600 of FIG. 6.

The example 600 may represent a logical block storage request where therequesting device directs the storage device 102 to store a data segmentat a particular physical address. The requesting device, such as aclient 110 or server 108, initiates a storage request to write data froma source data storage device. A portion of data 602 from the source datastorage device is depicted along with a data segment 606 stored in theportion of data 602. In this case, the data segment 606 is intended tobe stored on the storage device 102 at a physical address of sector 1,offset 5 and a data length of 8. In one embodiment, the requestingdevice formulates a storage request 608 that includes a header withsource parameters, including a logical block address and data length,and transmits the storage request 608 to the storage device 102.

For simplicity, this example 600 assumes that data of the data segment606 is not compressed. In this example 600, the data length of the datasegment 606 is 8 and there is room for a data packet in the current page616 where data is being stored for a data packet of data length 5. Inthis instance, the translation module 204 determines that the datasegment 606 will not fit at the end of a current page 616 where data iscurrently stored and creates two append data storage commands 614 fromthe data segment 606 to store two data packets, Data 1 610 and Data 2612.

The page 616 where data is being currently stored includes valid data618. In other pages may include valid and invalid data. One append datastorage command 614 stores Data 1 610 at Append Point 1 620, which isjust after a location where data was most recently stored 622. Datapacket Data 1 610 is then stored at the end of the current page 616 asshown 624.

A second append data storage command 614 stores Data 2 612 in the nextpage 626 where data is to be stored. In one embodiment, the next page626 is in a different erase block than the page 616 where Data 1 610 isstored. In this embodiment, data stored at Append Point 1 620 may flowto a next page without having to set a new append point at the beginningof the next page unless the next page 626 is in another erase block. Inanother embodiment, the next page 626 is a page adjacent to the page 616where Data 1 610 is stored or is somehow logically a next page, but anew Append Point 2 630 is required at the beginning of the next page626. One of skill in the art will recognize when a second append point630 is required to continue storing data. The next page 626 contains novalid data 628, either because the next page 626 has been erased or astorage space recovery process has determined that there is no longervalid data in the next page 626. The second append data storage command614 stores the data packet Data 2 612 at Append Point 2 630 as shown632.

While this example 600 is indicative of a case where a data segment 606is split because data packets 610, 612 created from the data segment 606falls on a page 616 boundary, in other cases data of the data segment606 maybe stored together, or may be split into three or more datapackets. In other cases, the compression module 306 compresses data ofthe data segment 606 to form one or more data packets 610, 612.

FIG. 7 is a schematic block diagram illustrating one embodiment of anapparatus 700 to efficiently map physical and virtual addresses inaccordance with the present invention. The apparatus 700 includes aforward mapping module 702, a reverse mapping module 704, and a storagespace recovery module 706, which are described below. At least a portionof one or more of the forward mapping module 702, the reverse mappingmodule 704, and the storage space recovery module 706 is located withinone or more of a requesting device that transmits the storage request,the data storage device 106, the storage controller 104, and a computingdevice separate from the requesting device, the data storage device 106,and the storage controller 104.

In one embodiment, the forward mapping module 702 and the reversemapping module 704 work in conjunction with the mapping module 206. Theforward mapping module 702 and the reverse mapping module 704 may bepart of the mapping module 206 or may be separate and work together withthe mapping module 206.

The apparatus 700 includes a forward mapping module 702 that uses aforward map to identify one or more physical addresses of data of a datasegment. The physical addresses are identified from one or more virtualaddresses of the data segment, which are identified in a storage requestdirected to the data storage device 106. For example, a storage requestmay include a request to read data stored in the data storage device106. The storage request to read data includes a virtual address orvirtual identifier associated with the data stored on the data storagedevice 106. The read request may include virtual address of a file fromwhich the data segment originated, which may be interpreted that theread request is a request to read an entire data segment associated withthe virtual address.

The read request, in another example, includes a virtual address alongwith an offset as well a data length of the data requested in the readrequest. For example, if a data segment is 20 blocks, a read request mayinclude an offset of 16 blocks (i.e. start at block 16 of 20) and a datalength of 5 so that the read request reads the last 5 blocks of the datasegment. The read request may include an offset and data length also ina request to read an entire data segment or to read from the beginningof a data segment. Other requests may also be included in a storagerequest, such as a status request. Other types and other forms ofstorage requests are contemplated within the scope of the presentinvention and will be recognized by one of skill in the art.

The apparatus 700 includes a forward map that maps of one or morevirtual addresses to one or more physical addresses of data stored inthe data storage device 106. The virtual addresses correspond to one ormore data segments relating to the data stored in the data storagedevice 106. The one or more virtual addresses typically include discreteaddresses within a virtual address space where the virtual addressessparsely populate the virtual address space. For a virtual address of adata segment, data length information may also be associated with thevirtual address and may also be included in the forward map. The datalength typically corresponds to the size of the data segment. Combininga virtual address and data length information associated with thevirtual address may be used to facilitate reading a particular portionwithin a data segment.

Often virtual addresses used to identify stored data represent a verysmall number of virtual addresses that are possible within a name spaceor range of possible virtual addresses. Searching this sparselypopulated space may be cumbersome. For this reason, the forward map istypically a data structure that facilitates quickly traversing theforward map to find a physical address based on a virtual address. Forexample, the forward map may include a B-tree, a content addressablememory (“CAM”), a binary tree, a hash table, or other data structurethat facilitates quickly searching a sparsely populated space or range.By using a forward map that quickly searches a sparsely populatedvirtual namespace, the apparatus 700 provides an efficient way todetermine one or more physical addresses from a virtual address.

While the forward map may be optimized, or at least designed, forquickly determining a physical address from a virtual address, typicallythe forward may is not optimized for locating all of the data within aspecific region of the data storage device 106. For this reason, theapparatus 700 includes a reverse mapping module 704 that uses a reversemap to determine a virtual address of a data segment from a physicaladdress. The reverse map is used to map the one or more physicaladdresses to one or more virtual addresses and can be used by thereverse mapping module 704 or other process to determine a virtualaddress from a physical address. The reverse map beneficially maps thedata storage device 106 into erase regions such that a portion of thereverse map spans an erase region of the data storage device 106 erasedtogether during a storage space recovery operation. The storage spacerecovery operation (or garbage collection operation) recovers eraseregions for future storage of data. By organizing the reverse map byerase region, the storage space recovery module 706 can efficientlyidentify an erase region for storage space recovery and identify validdata. The storage space recovery module 706 is discussed in more detailbelow.

The physical addresses in the reverse map are associated or linked withthe forward map so that if virtual address A is mapped to physicaladdress B in the forward map, physical address B is mapped to virtualaddress A in the reverse map. In one embodiment, the forward mapincludes physical addresses that are linked to entries in the reversemap. In another embodiment, the forward map includes pointers tophysical addresses in the reverse map or some other intermediate list,table, etc. One of skill in the art will recognize other ways to linkphysical addresses to the forward map and reverse map.

In one embodiment, the reverse map includes one or more sourceparameters. The source parameters are typically received in conjunctionwith a storage request and include at least one or more virtualaddresses. The source parameters may also include data lengthsassociated with data of a data segment received in conjunction with astorage request. In another embodiment, the reverse map does not includesource parameters in the form of virtual addresses or data lengths andthe source are stored with data of the data segment stored on the datastorage device 106. In this embodiment, the source parameters may bediscovered from a physical address in the reverse map which leads to thesource parameters stored with the data. Said differently, the reversemap may use the primary virtual-to-physical map rather than thesecondary-logical-to-physical map.

Storing the source parameters with the data is advantageous in asequential storage device because the data stored in the data storagedevice 106 becomes a log that can be replayed to rebuild the forward andreverse maps. This is due to the fact that the data is stored in asequence matching when storage requests are received, and thus thesource data serves a dual role; rebuilding the forward and reverse mapsand determining a virtual address from a physical address.

The apparatus 700 includes a storage space recovery module 706 that usesthe reverse map to identify valid data in an erase region prior to anoperation to recover the erase region. The identified valid data ismoved to another erase region prior to the recovery operation. Byorganizing the reverse map by erase region, the storage space recoverymodule 706 can scan through a portion of the reverse map correspondingto an erase region to quickly identify valid data or to determine aquantity of valid data in the erase region. An erase region may includean erase block, a fixed number of pages, etc. erased together. Thereverse map may be organized so that once the entries for a particularerase region are scanned, the contents of the erase region are known.

By organizing the reverse map by erase region, searching the contents ofan erase region is more efficient than searching a B-tree, binary tree,or other similar structure used for virtual-to-physical addresssearches. Searching forward map in the form of a B-tree, binary tree,etc. is cumbersome because the B-tree, binary tree, etc. wouldfrequently have to be searched in its entirety to identify all of thevalid data of the erase region. The reverse may include a table, database, or other structure that allows entries for data of an erase regionto be stored together to facilitate operations on data of an eraseregion.

In one embodiment, the forward map and the reverse map are independentof a file structure, a name space, a directory, etc. that organize datafor the requesting device transmitting the storage request, such as afile server or client operating in the server 108 or client 110. Bymaintaining the forward map and the reverse map separate from any fileserver of the requesting device, the apparatus 700 is able to emulate arandom access, logical block storage device storing data as requested bythe storage request.

Use of the forward map and reverse map allows the apparatus 700 toappear to be storing data in specific locations as directed by a storagerequest while actually storing data sequentially in the data storagedevice 106. Beneficially, the apparatus 700 overcomes problems thatrandom access causes for solid-state storage, such as flash memory, byemulating logical block storage while actually storing datasequentially. The apparatus 700 also allows flexibility because onestorage request may be a logical block storage request while a secondstorage request may be an object storage request, file storage request,etc. Maintaining independence from file structures, namespaces, etc. ofthe requesting device provides great flexibility as to which type ofstorage requests may be serviced by the apparatus 700.

FIG. 8 is a schematic block diagram illustrating another embodiment ofan apparatus 800 for efficient mapping of virtual and physical addressesin accordance with the present invention. The apparatus 800 includes aforward mapping module 702, a reverse mapping module 704, and a storagespace recovery module 706, which are substantially similar to thosedescribed above in relation to the apparatus 200 of FIG. 7. Theapparatus 800 also includes a map rebuild module 802, a checkpointmodule 804, a map sync module 806, an invalidate module 808, and a mapupdate module 810, which are described below.

The apparatus 800 includes a map rebuild module 802 that rebuilds theforward map and the reverse map using the source parameters stored withthe data. Where data is stored on the data storage device 106sequentially, by keeping track of the order in which erase regions orerase blocks in the data storage device 106 were filled and by storingsource parameters with the data, the data storage device 106 becomes asequential log. The map rebuild module 802 replays the log bysequentially reading data packets stored on the data storage device 106.Each physical address and data packet length is paired with the sourceparameters found in each data packet to recreate the forward and reversemaps.

In another embodiment, the apparatus 800 includes a checkpoint module804 that stores information related to the forward map and the reversemap where the checkpoint is related to a point in time or state of thedata storage device. The stored information is sufficient to restore theforward map and the reverse map to a status related to the checkpoint.For example, the stored information may include storing the forward andreverse maps in non-volatile storage, such as on the data storagedevice, along with some identifier indicating a state or timecheckpoint.

For example, a timestamp could be stored with the checkpointinformation. The timestamp could then be correlated with a location inthe data storage device 106 where data packets were currently beingstored at the checkpoint. In another example, state information isstored with the checkpoint information, such as a location in the datastorage device 106 where data is currently being stored. One of skill inthe art will recognize other checkpoint information that may be storedby the checkpoint module 804 to restore the forward and reverse maps tothe checkpoint.

In another embodiment, the apparatus 800 includes a map sync module 806that updates the forward map and the reverse map from the status relatedto the checkpoint to a current status by sequentially applying sourceparameters and physical addresses. The source parameters applied arestored with data that was sequentially stored after the checkpoint. Thephysical addresses are derived from a location of the data on the datastorage device 106.

Beneficially the map sync module 806 restores the forward and reversemaps to a current state from a checkpoint rather than starting fromscratch and replaying the entire contents of the data storage device106. The map sync module 806 uses the checkpoint to go to the datapacket stored just after the checkpoint and then replays data packetsfrom that point to a current state where data packets are currentlybeing stored on the data storage device 106. The map sync module 806typically takes less time to restore the forward and reverse maps thanthe map rebuild module 802.

In one embodiment, the forward and reverse maps are stored on the datastorage device 106 and another set of forward and reverse maps arecreated to map the stored forward and reverse maps. For example, datapackets may be stored on a first storage channel while the forward andreverse maps for the stored data packets may be stored as data on asecond storage channel; the forward and reverse maps for the data on thesecond storage channel may be stored as data on a third storage channel,and so forth. This recursive process may continue as needed foradditional forward and reverse maps. The storage channels may be on asingle data storage device 106 or on separate data storage devices 106.

The apparatus 800 includes an invalidate module 808 that marks an entryfor data in the reverse map indicating that data referenced by the entryis invalid in response to an operation resulting in the data beinginvalidated. The invalidate module 808 may mark an entry invalid as aresult of a delete request, a read-modify-write request, and the like.The reverse map includes some type of invalid marker or tag that may bechanged by the invalidate module 808 to indicate data associated with anentry in the reverse map is invalid. For example, the reverse map mayinclude a bit that is set by the invalidate module 808 when data isinvalid.

In one embodiment, the reverse map includes information for valid dataand invalid data stored in the data storage device 106 and the forwardincludes information for valid data stored in the data storage device106. Since the reverse map is useful for storage space recoveryoperations, information indicating which data in an erase block isinvalid is included in the reverse map. By maintaining the informationindicating invalid data in the reverse map, the forward map, in oneembodiment, need only maintain information related to valid data storedon the data storage device 106, thus improving the efficiency and speedof forward lookup.

The storage space recovery module 706 may then use the invalid marker todetermine a quantity of invalid data in an erase region by scanning thereverse map for the erase region to determine a quantity of invalid datain relation to a storage capacity of the erase region. The storage spacerecovery module 706 can then use the determined quantity of invalid datain the erase region to select an erase region for recovery. By scanningseveral erase regions, or even all available erase regions, the storagespace recovery module 706 can use selection criteria, such as highestamount of invalid data in an erase region, to then select an eraseregion for recovery.

Once an erase region is selected for recovery, in one embodiment thestorage space recovery module 706 may then write valid data from theselected erase region to a new location in the data storage device 106.The new location is typically within a page of an erase region wheredata is currently being stored sequentially. The storage space recoverymodule 706 may write the valid data using a data pipeline as describedin U.S. patent application Ser. No. 11/952,091 entitled “Apparatus,System, and Method for Managing Data Using a Data Pipeline” for DavidFlynn et al. and filed Dec. 6, 2007, which is hereinafter incorporatedby reference.

In one embodiment, the storage space recovery module 706 also updatesthe reverse map to indicate that the valid data written to the newlocation is invalid in the selected erase region and updates the forwardand reverse maps based on the valid data written to the new location. Inanother embodiment, the storage space recovery module 706 coordinateswith the map update module 810 (described below) to update the forwardand reverse maps.

In a preferred embodiment, the storage space recovery module 706operates autonomously with respect to data storage and retrievalassociated with storage requests and other commands. Storage spacerecovery operations that may be incorporated in the storage spacerecovery module 706 are described in more detail in the Storage SpaceRecovery Application referenced above.

In one embodiment, the apparatus 800 includes a map update module 810that updates the forward map and/or the reverse map in response tocontents of the data storage device 106 being altered. In a furtherembodiment, the map update module 810 receives information linking aphysical address of stored data to a virtual address from the datastorage device based on a location where the data storage device storedthe data. In the embodiment, the location where a data packet is storedmay not be available until the data storage device 106 stores the datapacket.

For example, where data from a data segment is compressed to form a datapacket, the size of each data packet may be unknown until aftercompression. Where the data storage device 106 stores data sequentially,once a data packet is compressed and stored, an append point is set to alocation after the stored data packet and a next data packet is stored.Once the append point is known, the data storage device 106 may thenreport back the physical address corresponding to the append point wherethe next data packet is stored. The map update module 810 uses thereported physical address and associated data length of the stored datapacket to update the forward and reverse maps. One of skill in the artwill recognize other embodiments of a map update module 810 to updatethe forward and reverse maps based on physical addresses and associateddata lengths of data stored on the data storage device 106.

FIG. 9 is a schematic flow chart diagram illustrating one embodiment ofa method 900 for efficient mapping of virtual and physical addresses inaccordance with the present invention. The method 900 begins and theforward mapping module 702 uses 902 the forward map to identify one ormore physical addresses of data of a data segment. The physicaladdresses are identified from one or more virtual addresses of the datasegment and the data segment is identified in a storage request directedto the data storage device 106. The forward map includes a mapping ofone or more virtual addresses to one or more physical addresses of datastored in the data storage device 106. The virtual addresses arediscrete addresses within a virtual address space where the virtualaddresses sparsely populate the virtual address space.

The reverse mapping module 704 uses 904 the reverse map to determine avirtual address of a data segment from a physical address. The reversemap maps one or more physical addresses to one or more virtualaddresses. The physical addresses in the reverse map are also associatedwith the forward map using pointers, links, etc. The virtual addressesin the reverse map correspond to one or more data segments relating tothe data stored in the data storage device 106. The reverse map alsomaps the data storage device into erase regions such that a portion ofthe reverse map spans an erase region. An erase region of the datastorage device 106 is erased together during a storage space recoveryoperation. The storage space recovery operation recovers erase regionsfor future storage of data.

The storage space recovery module 706 uses 906 the reverse map toidentify valid data in an erase region prior to an operation to recoverthe erase region and the method 900 ends. The storage space recoverymodule 706 or other module associated with storage space recovery movesthe identified valid data to another erase region prior to the recoveryoperation. Note that the steps 902, 904, 906 of the method 900 are shownin parallel because the steps 902, 904, 906 may be practicedindependently in any order.

FIG. 10 is a schematic flow chart diagram illustrating anotherembodiment of a method 1000 for efficient mapping of virtual andphysical addresses in accordance with the present invention. The method1000 begins and the storage space recovery module 706 determines 1002 aquantity of invalid data in an erase region by scanning the reverse mapfor the erase region to determine a quantity of invalid data in relationto a storage capacity of the erase region. The storage space recoverymodule 706 then determines 1004 if there is another erase region toevaluate. If the storage space recovery module 706 determines 1004 thereis another erase region to evaluate, the storage space recovery module706 determines a quantity of invalid data for the next erase region.

If the storage space recovery module 706 determines 1004 there is notanother erase region to evaluate, the storage space recovery module 706selects 1006 an erase region for recovery by using selection criteria,which may include using the quantity of invalid data in an erase region.The storage space recovery module 706 identifies 1008 valid data in theselected erase region and moves 1010 the valid data to an erase regionwhere data is currently being written. The map update module 810 thenupdates 1012 the forward and reverse maps to reflect that the valid datahas been written to another location in the data storage device 106.

In one embodiment, the storage space recovery module 706 erases 1014 theselected erase region and marks 1014 the selected storage regionavailable for data storage and the method 1000 ends. In anotherembodiment, once the storage space recovery module 706 has written allvalid data in the selected erase region to another location, the storagespace recovery module 706 marks 1014 the selected storage regionavailable for data storage without erasure.

FIG. 11 is a schematic block diagram of an example of a forward map anda reverse map in accordance with the present invention. Typically, theapparatus 700, 800 receives a storage request, such as storage requestto read and address. For example, the apparatus 700, 800 may receive alogical block storage request 1102 to start reading read address 182 andread 3 blocks. Typically the forward map 1104 stores logical blockaddresses as virtual addresses along with other virtual addresses so theforward mapping module 702 uses forward map 1104 to identify a physicaladdress from the virtual address 182 of the storage request 1102. In theexample, for simplicity only virtual addresses that are numeric areshown, but one of skill in the art will recognize that any virtualaddress may be used and represented in the forward map 1104. A forwardmap 1104, in other embodiments, may include alpha-numerical characters,hexadecimal characters, and the like.

In the example, the forward map 1104 is a simple B-tree. In otherembodiments, the forward map 1104 may be a content addressable memory(“CAM”), a binary tree, a hash table, or other data structure known tothose of skill in the art. In the embodiment, a B-Tree includes nodes(e.g. the root node 1108) that may include two virtual addresses. Eachvirtual address may be a range. For example, a virtual address may be inthe form of a virtual identifier with a range (e.g. offset and length)or may represent a range using a first and a last address or location.

Where a single virtual address is included at a particular node, such asthe root node 1108, if a virtual address 1106 being searched is lowerthan the virtual address of the node, the search will continue down adirected edge 1110 to the left of the node 1108. If the searched virtualaddress 1106 matches the current node 1108 (i.e. is located within therange identified in the node), the search stops and the pointer, link,physical address, etc. at the current node 1108 is identified. If thesearched virtual address 1106 is greater than the range of the currentnode 1108, the search continues down directed edge 1112 to the right ofthe current node 1108. Where a node includes two virtual addresses and asearched virtual address 1106 falls between the listed virtual addressesof the node, the search continues down a center directed edge (notshown) to nodes with virtual addresses that fall between the two virtualaddresses of the current node 1108. A search continues down the B-treeuntil either locating a desired virtual address or determining that thesearched virtual address 1106 does not exist in the B-tree.

In the example depicted in FIG. 11, the forward mapping module 702searches for virtual address 182 1106 starting at the root node 1108.Since the searched virtual address 1106 is lower than the virtualaddress in the root node, 2005-212, the forward mapping module 702searches down the directed edge 1110 to the left to the next node 1114.The searched virtual address 182 1106 more than the virtual address(072-083) stored in the next node 1114 so the forward mapping module 702searches down a directed edge 1116 to the right of the node 1114 to thenext node 1118. In this example, the next node 1118 includes a virtualaddress of 178-192 so that the searched virtual address 182 1106 matchesthe virtual address 178-192 of this node 1118 because the searchedvirtual address 182 1106 falls within the range 178-192 of the node1118.

Once the forward mapping module 702 determines a match in the forwardmap 1104, the forward mapping module 702 returns a physical address,either found within the node 1118 or linked to the node 1118. In thedepicted example, the node 1118 identified by the forward mapping module702 as containing the searched virtual address 1106 includes a link “f”that maps to an entry 1120 in the reverse map 1122.

In the depicted embodiment, for each entry 1120 in the reverse map 1122(depicted as a row in a table), the reverse map 1122 includes an entryID 1124, a physical address 1126, a data length 1128 associated with thedata stored at the physical address 1126 on the data storage device 106(in this case the data is compressed), a valid tag 1130, a virtualaddress 1132 (optional), a data length 1134 (optional) associated withthe virtual address 1132, and other miscellaneous data 1136. The reversemap 1122 is organized into erase blocks (erase regions). In thisexample, the entry 1120 that corresponds to the selected node 1118 islocated in erase block n 1138. Erase block n 1138 is preceded by eraseblock n−1 1140 and followed by erase block n+1 1142 (the contents oferase blocks n−1 and n+1 are not shown). An erase block may be someerase region that includes a predetermined number of pages. An eraseregion is an area in the data storage device 106 erased together in astorage recovery operation,

While the entry ID 1124 is shown as being part of the reverse map 1122,the entry ID 1124 may be an address, a virtual link, or other means totie an entry in the reverse map 1122 to a node in the forward map 1104.The physical address 1126 is an address in the data storage device 106where data that corresponds to the searched virtual address 1106resides. The data length 1128 associated with the physical address 1126identifies a length of the data packet stored at the physical address1126. (Together the physical address 1126 and data length 1128 may becalled destination parameters 1144 and the virtual address 1132 andassociated data length 1134 may be called source parameters 1146 forconvenience.) In the example, the data length 1120 of the destinationparameters 1144 is different from the data length 1134 of the sourceparameters 1146 in one embodiment compression the data packet stored onthe data storage device 106 was compressed prior to storage. For thedata associated with the entry 1120, the data was highly compressibleand was compressed from 64 blocks to 1 block.

The valid tag 1130 indicates if the data mapped to the entry 1120 isvalid or not. In this case, the data associated with the entry 1120 isvalid and is depicted in FIG. 11 as a “Y” in the row of the entry 1120.Typically the reverse map 1122 tracks both valid and invalid data andthe forward map 1104 tracks valid data. In the example, entry “c” 1148indicates that data associated with the entry 1148 is invalid. Note thatthe forward map 1104 does not include virtual addresses. The reverse map1122 typically maintains entries for invalid data so that valid andinvalid data can be quickly distinguished during a storage recoveryoperation.

The depicted reverse map 1122 includes source parameters 1146 forconvenience, but the reverse map 1122 may or may not include the sourceparameters 1146. For example, if the source parameters 1146 are storedwith the data, possibly in a header of the stored data, the reverse map1122 could identify a virtual address indirectly by including a physicaladdress 1126 associated with the data and the source parameters 1146could be identified from the stored data. One of skill in the art willrecognize when storing source parameters 1146 in a reverse map 1122would be beneficial.

The reverse map 1122 may also include other miscellaneous data 1136,such as a file name, object name, source data, etc. One of skill in theart will recognize other information useful in a reverse map 1122. Whilephysical addresses 1126 are depicted in the reverse map 1122, in otherembodiments, physical addresses 1126, or other destination parameters1144, may be included in other locations, such as in the forward map1104, an intermediate table or data structure, etc.

Typically, the reverse map 1122 is arranged by erase block or eraseregion so that traversing a section of the map associated with an eraseblock (e.g. erase block n 1138) allows the storage space recovery module706 to identify valid data in the erase block 1138 and to quantify anamount of valid data, or conversely invalid data, in the erase block1138. Arranging an index into a forward map 1104 that can be quicklysearched to identify a physical address 1126 from a virtual address 1106and a reverse map 1122 that can be quickly searched to identify validdata and quantity of valid data in an erase block 1138 is beneficialbecause the index may be optimized for searches and storage recoveryoperations. One of skill in the art will recognize other benefits of anindex with a forward map 1104 and a reverse map 1122.

FIG. 12 is a schematic block diagram illustrating one embodiment of anapparatus 1200 for coordinating storage requests in accordance with thepresent invention. The apparatus 1200 includes a storage controller 104with an append/invalidate module 1202 and a restructure module 1204,which are described below. At least a portion of one or more of theappend/invalidate module 1202 and the restructure module 1204 is locatedwithin one or more of a requesting device that transmits the storagerequest, the data storage device 106, the storage controller 104, and acomputing device separate from the requesting device, the data storagedevice 106, and the storage controller 104.

The apparatus 1200 includes an append/invalidate module 1202 thatgenerates a first append data storage command in response to receiving afirst storage request and that generates a second append data storagecommand in response to receiving a second storage request. The first andsecond storage requests are received from one or more requestingdevices. A requesting device may be the server 108 or a client 110 onthe server 108 or in communication with the server 108 over a computernetwork 112.

The first storage request includes a request to overwrite existing dataof a data segment with first data. The data segment is stored on a datastorage device 106. The second storage request includes a request tooverwrite existing data of the same data segment with second data. Thefirst and second data include at least a portion of overlapping data tobe stored at a common offset within the data segment and the secondstorage request is received after the first storage request.

The append/invalidate module 1202 also updates an index in response tothe first storage request by marking data of the data segment invalid.The data that is marked invalid is data being replaced by the firstdata. The append/invalidate module 1202 also updates the index inresponse to the second storage request by marking data of the datasegment invalid where the data marked invalid is data being replaced bythe second data. In one embodiment, the append/invalidate module 1202updates the index by updating a reverse map 1122. In one embodiment, theappend/invalidate module 1202 updates the reverse map 1122 to indicatethat the data segment is invalid. In another embodiment, theappend/invalidate module 1202 marks only the data corresponding to thefirst data or to the second data invalid. This may require modifying theforward and reverse maps 1104, 1122 and is explained in detail below.

In another embodiment, marking data of the data segment as invalid mayinclude generating a list indicating portions of the data segment beingreplaced by the first data and the second data are invalid and alsoindicating that portions of the data segment not being replaced arestill valid. The list may be used by the restructure module 1204 inupdating the index without marking the entire data segment invalid.

The apparatus 1200 includes a restructure module 1204 that updates theindex based on the first data and updates the index based on the seconddata, where the updated index is organized to indicate that the seconddata is more current than the first data. This organization of the indexis maintained when either the index is updated based on the first databefore being updated based on the second data or the index is updatedbased on the second data before being updated based on the first data.

In one embodiment, organizing the index to indicate that for theoverlapping data the second data is more current than the first dataincludes maintaining a pointer, link, etc. in the index for theoverlapping data that corresponds to the second data regardless of theorder of the update. For example, if the original data segment includesblock 3 that is overwritten by the first data with 3′ and by the seconddata with 3″, the index points to 3″ after the updates regardless ofwhether or not the index is updated with the second data before thefirst data or vice versa. Further explanation is provided below in thedescription of FIG. 16.

Organizing the index to indicate that the second data is more currentthan the first data may typically includes updating a portion of theindex mapped to the overlapping data with the second data and retainingthe mapping to the second data even in cases where the restructuremodule 1204 updates the index based on the second data before updatingthe index based on the first data.

While the append/invalidate module 1202 is updating the index, theappend/invalidate module 1202 prevents access to index by anotherprocess or module to ensure data integrity. For example, if theappend/invalidate module 1202 is updating the index based on the firstdata, the append/invalidate module 1202 prevents another instance of theappend/invalidate module 1202 or the restructure module 1204 fromupdating the index based on the second data. By preventing access to theindex (i.e. locking the index) while the index is updated, the apparatus1200 supports multiple instances of the modules 1202, 1204 of theapparatus running on multiple processors or in multiple threads. Notethat while two storage requests are discussed above, the presentinvention applies equally to situations where three or more storagerequests are processed simultaneously.

FIG. 13 is a schematic block diagram illustrating another embodiment ofan apparatus 1300 for coordinating storage requests in accordance withthe present invention. The apparatus 1300 includes an append/invalidatemodule 1202 an a restructure module 1204 which are substantially similarto those described in relation to the apparatus 1200 of FIG. 12. Theapparatus 1300 also includes a data location module 1302, a read module1304, a read delay module 1306, and a sequence number module 1308, whichare described below. The modules 1202, 1204, 1302-1308 of the apparatus1300 are depicted in a storage controller 104, but all or a portion ofthe modules 1202, 1204, 1302-1308 may be included in the data storagedevice 106, client 110, server 108, or other device or location.

In one embodiment, the apparatus 1300 includes a data location module1302 that updates the index with a first physical location where thedata storage device 106 stored the first data and also updates the indexwith a second physical location where the data storage device 106 storedthe second data. The apparatus 1300 receives the physical locationswhere the data storage device 106 stored the first data and the seconddata received from the data storage device 106. As discussed above,where the data storage device 106 stores data sequentially, the locationwhere data is stored for a given data packet may not be known until aprevious data packet is stored.

The updated index is organized to indicate that the second physicallocation is more current than the first physical location regardless ofwhether the index is updated based on the first physical location beforebeing updated based on the second physical location or the index isupdated based on the second physical location before being updated basedon the first physical location. For example, if an overlapping portionof the first data is stored at Address 1 in response to the firststorage request and an overlapping portion of the second data is storedat Address 2, the data location module 1302 stores Address 2 in theindex for the portion of the index relating to the overlapping data. Ifthe index is updated for the second data before updating for the firstdata, the index maintains Address 2 for the overlapping data rather thanreplacing Address 2 with Address 1.

The data location update module 1302 prevents access to the index byanother instance of the data location module 1302 or another module,such as the append/invalidate module 1202 or the restructure module1204, until the data location module 1302 has completed updating theindex. Preventing access to the index while the data location module1302 is updating the index provides increased data reliability.

Typically, processing of a particular storage request by theappend/invalidate module 1202 precedes processing of the storage requestby the restructure module 1204 and processing of the storage request bythe restructure module 1204 precedes processing of the storage requestby the data location module 1302. However, once the order of arrival oftwo storage requests has been determined, processing of a second storagerequest by an instance of the append/invalidate module 1202 may occurbefore processing of the first storage request by another instance ofthe append/invalidate module 1202. Likewise, processing of the secondstorage request by the restructure module 1204 may occur beforeprocessing of the first storage request by the append/invalidate module1202 or by another instance of the restructure module 1204.

Similarly, processing of the second storage request by an instance ofthe data location module 1302 may occur before processing of the firststorage request by the append/invalidate module 1202, the restructuremodule 1204, or another instance of the data location module 1302. Thisfeature of the present invention allows asynchronous and independentmulti-processor and/or multi-thread processing of storage requests.Preventing access to a module 1202, 1204, 1302 while another instance ofthe module 1202, 1204, 1302 is updating the index and organizing theindex base on order of arrival rather than order of processingfacilitates processing storage requests in parallel by multipleprocessors or threads.

The apparatus 1300 includes a reverse read module 1304 that reads atleast a portion of the data segment in response to a storage requestthat includes a read request. Read requests must be coordinated withstorage requests that result in modification of a data segment so theapparatus 1300 also includes a read delay module 1306 that delaysservicing the requested read until the first storage request is servicedby the append/invalidate module 1202, the restructure module 1204, andthe data location update module 1302. The read delay module 1306maintains data integrity by preventing a read while contents of a datasegment are updated or while the index mapped to the data segment isupdated.

In one embodiment, when a read request to read a data segment isreceived after a first storage request that overwrites at least aportion of the data segment but before a second storage request thatalso overwrites at least a portion of the data segment, the read delaymodule 1306 delays servicing the read request until both the first andsecond storage requests are serviced by the append/invalidate module1202, the restructure module 1204, and the data location update module1302. In this embodiment, the read delay module 1306 allows the datasegment to be updated based on the second data storage request so thatthe read module 1304 will read the most current version of the datasegment.

The apparatus 1300 includes, in one embodiment, a sequence number module1308 that associates a sequence number with a storage request where theassigned sequence numbers represent an order that storage requests arereceived by the apparatus 1300. Sequence numbers facilitate organizingthe index so that the index reflects that an update based on a secondstorage request takes precedent over an update based on a first storagerequest. In this embodiment, the restructure module 1204 organizes theindex to indicate that the second data is more current than the firstdata by using a sequence number assigned to each of the first storagerequest and the second storage request. Likewise, the data locationmodule 1302 also organizes the index to indicate that the second data ismore current than the first data. The sequence number may be a timestamp, a number in a series, or any other mechanism that may be used toidentify that one sequence number precedes another sequence number. Oneof skill in the art will recognize other forms of a sequence number.

In one embodiment, an instance append/invalidate module 1202, therestructure module 1204, or the data location module 1302 does notprevent access to the entire index while updating the index. The indexmay be divided into two or more regions. For example, one region maypertain to one area of the data storage device 106 and another region ofthe index may pertain to another area of the data storage device 106. Inthe embodiment, while a storage request pertaining to a first region ofthe index is serviced, additional instances of the append/invalidatemodule 1202, the restructure module 1204, or the data location module1302 may service a second storage request pertaining to the secondregion. In another embodiment, the index may be divided to createmultiple, virtual address regions that may be operated on independently.A region of the index may be a branch, a sub-branch, or even a node aslong as restricting access to a region while the region is updated doesnot affect data integrity of other regions being updated simultaneously.

In one embodiment, the first and second storage requests are receivedwith data that will replace at least a portion of the data segment. Inanother embodiment, one or both of the first and second storage requestsare received substantially without data. In addition, correspondingappend data storage requests may be transmitted to the data storagedevice 106 without data. For example, a storage request may not includedata and may initiate either a direct memory access (“DMA”) process or aremote DMA (“RDMA”) process to transfer data of the data segment to thedata storage device 106. Likewise, an append data storage commandtransmitted to the data storage device 106 may direct the data storagedevice 106 to set up a DMA or RDMA process to transfer data. Theapparatus 1200, 1300 is flexible enough to handle one storage requestwith data, another that is part of a recovery process, and another thatsets up a DMA or RDMA operation.

While the present invention discloses how instances of theappend/invalidate module 1202, the restructure module 1204, and the datalocation update module 1302 handle requests received at about the sametime and that affect a single data segment, one of skill in the art willrecognize that the append/invalidate module 1202, the restructure module1204, and the data location update module 1302 may handle a variety ofother storage requests that affect different portions of a single datasegment and also may handle storage requests affecting two or moreseparate data segments.

FIG. 14 is a schematic flow chart diagram illustrating one embodiment ofa method 1400 for coordinating storage requests in accordance with thepresent invention. The method 1400 begins and the apparatus 1200receives 1402 a first storage request. The apparatus 1200 receives 1404a second storage request. The first and second storage requests affect asingle data segment by overwriting at least a portion of the datasegment. In addition, the first and second data requests overwrite atleast one overlapping portion of the data segment common to both thefirst and second storage requests.

An instance of the append/invalidate module 1202 generates 1406 a firstappend data storage command to overwrite at least a portion of the datasegment with first data. An instance of the append/invalidate module1202 also generates 1408 a second append data storage command tooverwrite at least a portion of the data segment with second data. Theinstance of the append/invalidate module 1202 that is servicing thefirst data request also updates 1410 the index by invalidating data ofthe data segment replaced by the first data. The instance of theappend/invalidate module 1202 that is servicing the second data requestalso updates 1412 the index by invalidating data of the data segmentreplaced by the second data.

An instance of the restructure module 1204 updates 1414 the index basedon the first data. An instance of the restructure module 1204 alsoupdates 1416 the index based on the second data and the method 1400ends. While any instance of the append/invalidate module 1202 or therestructure module 1204 is updating the index, other instances of themodules 1202, 1204 are prevented from accessing the index.

The order of the steps 1406-1416 related to instances of theappend/invalidate module 1202 and the restructure module 1204 are merelyone embodiment. Other ordering of the steps 1406-1416 are possible andare an important feature of the present invention as long as the firststorage request is serviced by the append/invalidate module 1202 beforethe first storage request is serviced by the restructure module 1204 andthe second storage request is serviced by the append/invalidate module1202 before the second storage request is serviced by the restructuremodule 1204. For example, an instance of the append/invalidate module1202 and an instance of the restructure module 1204 may service thesecond storage request prior to another instance of theappend/invalidate module 1202 services the first storage request.Possible ordering of servicing storage requests is discussed in moredetail below with respect to the example depicted in FIG. 16.

FIG. 15 is a schematic flow chart diagram illustrating anotherembodiment of a method 1500 for coordinating storage requests inaccordance with the present invention. The method 1500 is an exampleshowing steps taken to coordinate storage requests in a client 110,storage controller 104 or other location of the apparatus 1200, 1300described above, and in the data storage device 106. Note that while themethod 1500 depicts action in three devices, the method 1500 is notintended to imply that the present invention is required to span morethan one device, nor is it intended to imply that the modules must belocated as shown in FIGS. 12 and 13.

The present invention may be practiced within the storage controller 104or other single device in communication with the storage controller 104of a data storage device 106 or may also include a portion of a driverwithin a client 110, server 108, etc. The actions depicted within theclient 110 and data storage device 106 are typically independent of thepresent invention and are shown merely to illustrate what typicallyhappens in the client 110 and data storage device 106 to transmit astorage request and to store data in response to the storage request.

While the method 1500 depicts a single client 110, multiple clients 110typically are present and each access the data storage device 106through one or more instances the storage controller 104. The method1500 begins when the client 110 initiates 1502 an inter-client lock thatcoordinates writing data with the other clients 110 so that the client110 shown is the only client 110 transmitting a storage request with awrite request at a particular time. The write request is a request maybe a request to write new data or to replace or modify an existing datasegment. While preventing access to the index is while servicing a writerequest to write new data is important, the present invention isparticularly useful when replacing or modifying an existing data segmentto ensure data integrity in the presence of two or more requests tomodify/replace the same data segment. In the embodiment, once theinter-client lock is in place to assure synchronization of writerequests from more than one client 110, the client 110 transmits 1504 awrite request to the storage controller 104. In another embodiment, nointer-client synchronization is utilized and storage requests have nopredetermined order. In the depicted embodiment, the write request issubstantially without data and initiates a DMA process to transfer datafrom the client 110 or other location to the data storage device 106.

The append/invalidate module 1202 then prevents multiple access to theindex by “locking” 1506 the index. The sequence module 1308 then gets1508 a sequence number and associates the sequence number with the writerequest. The append/invalidate module 1202 then creates 1510 an appenddata storage command based on the write request. The append data storagecommand includes the assigned sequence number and relates to data in thedata segment that is requested to be overwritten by the write request.The append/invalidate module 1202 or other module in the storagecontroller 104 then transmits 1511 the append data storage command tothe data storage device 106 with the sequence number 1511. The appenddata storage command includes a command to initiate a DMA process totransfer data from the client 110 or other location to the data storagedevice 106.

In addition, the append/invalidate module 1202 updates 1512 the index bymarking existing data as invalid. The existing data is data that is partof the data segment and is to be replaced by the data referenced in thewrite request. The append/invalidate module 1202 then releases 1514 thelock on the index so that other write requests may be serviced. Inaddition, the client 110 unlocks 1515 transmitting write requests sothat another client 110 can send a write request to the storagecontroller 104. In one embodiment, the append/invalidate module 1202invalidates the entire data segment temporarily until the index isupdated. After the index is updated, the portions of the data segmentnot affected by the write request are marked valid. In anotherembodiment, the append/invalidate module 1202 invalidates only theportion of the index associated with the data being overwritten.

Note that creating 1510 the append data storage command is shown inparallel to invalidating 1513 data. In another embodiment, theappend/invalidate module 1202 creates 1510 the append data storagecommand after invalidating 1512 data and releasing 1514 the lock on theindex. In the preferred embodiment, the data storage device 106 storesdata packets in order of the sequence numbers. By associating thesequence number 1511 with the append data storage command, theappend/invalidate module 1202 may, in one embodiment, create 1510 theappend data storage command independent of invalidating 1512 data andunlocking 1512 the index.

Once the append command and sequence number are received by the datastorage device 106, the data storage device 106 initiates 1516 a DMAprocess to transfer the data associated with the write request to thedata storage device 106. The data storage device 106 also blocks 1518reading of this data to be stored and processes 1520 the received data.Processing the data might include pre-pending headers, compressing thedata, encrypting the data, creating error correcting code (“ECC”), etc.Processing may also include multiple DMA transfers, though only one isshown. The data storage device 106 then completes the DMA for the appenddata storage command associated with the sequence number and thencompletes 1524 the append data storage command.

In other embodiments, the append/invalidate module 1202 creates othercommands associated with the write request in addition to the appenddata storage command and associates the commands with the sequencenumber. The storage controller 104 then transmits the commands with thesequence number attached to each command. The data storage device 106then completes 1524 all commands associated with the sequence number.One of skill in the art will recognize other commands associated with awrite request that may be generated to service the write request. Oncethe data storage device 106 processes 1520 the data received through theDMA, the data storage device 106 stores 1526 the data and unblocks 1528reading the data. Note that storing 1526 the data is a process that maytake longer than other processes. In the preferred embodiment, thestorage controller 104 will process many other storage requests whilethe data is being stored.

After the append/invalidate module 1202 unlocks the 1514 the index, therestructure module 1204 locks 1530 the index and updates 1532 the indexbased on the data requested to be written with respect to the writerequest received from the client 110. The restructure module 1204 thenunlocks 1534 the index. Once unlocked, the index can be accessed byother instances of the append/invalidate module 1202 and the restructuremodule 1204. In one embodiment, instead of the append/invalidate module1202 unlocking 1514 the index and the restructure module 1204 re-locking1530 the index, the index remains locked throughout the invalidateprocess of the append/invalidate module 1202 and the index update of therestructure module 1204.

Once the data storage device 106 completes 1524 the command(s)associated with a sequence number 1511, the data storage device 106transmits 1536 one or more physical addresses where the data storagedevice 106 stored 1526 the associated data. In one embodiment, thecommand completion 1524 is transmitted with a sequence number 1536. Inthe preferred embodiment, the sequence number 1536 transmitted with thecommand completion 1524 is the same as sequence number 1511 astransmitted initially to the data storage device 106. The data locationmodule 1302 locks 1538 the index and updates 1540 the index to point tothe location(s) where the data storage device 106 stored 1526 the data.In the embodiment, the append/invalidate module 1202 clears theinvalidate flag for the associated data segment. In another embodimentwhere the append/invalidate module 1202 invalidated the entire datasegment associated with the write request, the data location module 1302clears the invalidate flag on data that was not affected be the writerequest. The data location module 1302 then unlocks 1542 the index andthe storage controller 104 completes 1544 the write request and sends aconfirmation to the client 110. The client 110 then receives 1546 andprocesses the confirmation and the method 1500 ends.

In one embodiment, each time the index is locked 1506, 1530, 1538, theentire index is locked. In another embodiment, only a portion of theindex is locked. For example, a branch maybe locked or even a nodecorresponding to the data segment is locked.

FIG. 16 (which includes FIGS. 16A, 16B, and 16C) is a schematic blockdiagram illustrating an example 1600 of an apparatus 1200, 1300 forcoordinating storage requests in accordance with the present invention.In the example. 1600, a data segment in an original state 1602 isassumed to be stored on the data storage device 106 based on a previouswrite request to store the data segment. Also in the example, theapparatus 1200, 1300 receives two storage requests from two clients 110:Client A sends Request 1 and Client B sends Request 2. Both storagerequests seek to overwrite a portion of the data segment. Request 1 fromClient A is received before Request 2 from Client B.

A sequence in time 1604 as requests are received is shown thatrepresents how a client 110 would perceive changes to the data segmentif the storage controller 104 emulates the data storage device 106 as arandom access device. The original state 1602 of the data segmentincludes five blocks numbered 1-5. A block in the original state 1602 isshown with no cross hatching. Client A sends Request 1 associated withFirst Data that modifies blocks 2 and 3. The First Data 1606 shows thenew data in blocks 2 and 3 as 2′ and 3′ respectively. A block with theFirst Data 1606 is shown with horizontal cross hatching. Client A sendsRequest 2 associated with Second Data that modifies blocks 3 and 4. TheSecond Data 1608 shows the new data in blocks 3 and 4 as 3″ and 4″respectively. A block with the Second Data 1608 is shown with diagonalcross hatching running from top-left to bottom-right. A final state 1610is shown where the First Data and Second Data overwrite blocks 2, 3, and4.

Of particular note is that in the final state 1610, block 3 includes theSecond Data 3″. This is due to Request 2 arriving after Request 1. SinceRequest 2 is second in time to Request 1, Request 2 is considered to bemore current than Request 1. The example 1600 depicts several scenariosfor updating the index for the data segment such that the final state ofthe index is the same in each case and the Second Data is more currentthan the First Data.

In the example 1600, once a request is received it is processed in threesteps. First, an instance of the append/invalidate module 1202invalidates data in the data segment that will be replaced by data ofthe write request. Second, the index is assumed to have a forward map1104 in the form of a B-tree, binary tree, or similar structure and aninstance of the restructure module 1204 updates the index byrestructuring the tree. Third, the data location module 1302 uses one ormore locations of where the data storage device 106 stored the data toupdate the index. For example, the data location module 1302 may updatea reverse map 1122. The three actions for Request A are shown as A1, A2,and A3. The three actions for Request B are shown as B1, B2, and B3. Inanother embodiment, steps A1 and A2 may be combined into a single step.Similarly, A2 and A3 could be combined. Rearranging and combining thesteps or processes executed within the step are consistent with thebroad scope of the invention.

The example 1600 depicts a state of the tree for 10 different sequences(S1-S10) of updating the tree based on various combinations of theactions associated with Request A and Request B. Note that the finalstate for all of the 10 possible sequences is the same, which is thedesired result. The sequences S1-S10 all start with step A1. Thisassumes that the append/invalidate module 1202 assigns a sequence numberand invalidates data for Request 1 from client A right after it wasreceived. In another embodiment, assigning sequence numbers may bedecoupled from invalidating data. In this embodiment, where Request 1from client A is received before Request 2 from client B and one or moreinstances of the append/invalidate module 1202 initially assign asequence number to each request based on order of arrival, since eachrequest has an assigned sequence number, processing in step A1 toinvalidated data may be delayed and B1 may be processed first toinvalidate data. In this case, the final state of additional sequenceswhere B1 precedes A1 (not shown) will be the same as shown in sequencesS1-S10.

The first sequence S1 depicts an original state of a node in the treerepresenting blocks 1-5 of the data segment. (Note that the noderepresenting the original state has no cross hatching to match theoriginal state of the blocks shown above.) The node shows a range of 1-5(“1:5”) which corresponds to the blocks 1-5 being in an original state1602. The first step of the sequence S1 is for the append/invalidatemodule 1202 to invalidate the node of the data segment. This is depictedat step A1 where the node is shown invalidated and an A below the nodeindicating that Request A is being serviced to invalidate the data.Invalidating the range is depicted with a cross hatch that runs fromtop-right to bottom-left and a letter under the node indicating whichrequest invalidated the data.

Note that for simplicity, the entire range 1-5 is shown as invalidatedin the sequences S1-S10 shown in the example 1600. However, in apreferred embodiment, only the blocks affected by a request areinvalidated. During this invalidation process the append/invalidatemodule 1202 locks all or a portion of the index.

The second step of the sequence S1, shown as A2, is for the restructuremodule 1204 to restructure the tree to split blocks 2 and 3, which willpoint to 2′ and 3′. At this stage, the lock for blocks 1, 4, and 5 arereleased and the data remains accessible for reads in the originalstate. Blocks 2 and 3 are shown in the middle node of the tree asinvalid. The third step of the sequence S1, shown as A3, is for the datalocation module 1302 to update the tree based on locations of the wherethe blocks 2′ and 3′ were stored on the data storage device 106. This isdepicted as the middle node showing new blocks 2′ and 3′ and the nodehaving a horizontal cross hatch corresponding to a final state of theblocks at the end of step A3 as shown in the completion of the client Aupdate 1606 above. This may be accomplished by updating pointers in thenode of the forward map 1104 to point to one or more entries in thereverse map 1122 where the entries in the reverse map 1122 have thephysical addresses of the First Data. In this case, the reverse map 1122has physical addresses for 2′ and 3′.

The fourth step of the sequence S1, shown as B1, is for an instance ofthe append/invalidate module 1202 to invalidate nodes in the tree to beoverwritten by the Second Data 3″ and 4″. In this case, the second node(2′:3′) and third node (4:5) are affected so both nodes are depicted asinvalid (right to left cross hatching). The fifth step of the sequenceS1, shown as B2, is for the restructure module 1204 to restructure thetwo nodes to form a node pointing to 2′, a node pointing to 3″:4″, and anode pointing to block 5. The node pointing to 3″:4″ is shown asinvalid.

The sixth step of the sequence S1, shown as B3, corresponds to aninstance of the data location module 1302 updating the index withphysical address information where the data storage device 106 storedblocks 3″ and 4″. At this point the node representing 3″ and 4″ is shownin a final state with cross hatching running from left to right. Step B3indicates the portion of the tree representing blocks 1-5 in a finalstate where blocks 1 and 5 are unmodified, block 2 indicates that it hasbeen updated with the First Data 2′, and blocks 3 and 4 indicate thatthey have been updated with the Second Data 3″ and 4″. Note that block 3is properly shown with the Second Data 3″ and that the final states ofall of the depicted sequences S1-S10 are the same regardless of theorder of processing.

Sequence S2 is the same as sequence 51 except that the order ofprocessing steps A3 and B1 are switched. At step B1, the instance of therestructure module 1204 servicing Request B invalidates the two nodespointing to blocks 3 and 4. At step A3, the instance of the datalocation module 1302 servicing Request A updates the index withlocations where 2′ and 3′ were stored, but the second node (2″:3″) andthird node (4:5) remain invalid because the instance of the restructuremodule 1204 servicing Request B has not yet separated out 3 and 4 fromthe second and third nodes. Once the restructure module 1204 hasrestructured the nodes to include a node pointing to 3″:4″ leaving anode pointing to 2′ and a node point to 5, the nodes pointing to 2′ and5 are in a final state. After step B3, which is the same as step B3 insequence 51, the nodes of sequence S2 are in a final state and match thefinal state of sequence S1.

Sequences S3-S10, shown on FIGS. 16B and 16C, follow a procedure andanalysis similar to that of sequences S1 and S2 except the steps arereordered. Note that in all sequences S1-S10, the final state is thesame. The present invention beneficially allows multiple processors ormultiple threads to access the same data segment stored on a common datastorage device 106 or a data segment striped across an array of datastorage devices 106. Beneficially, the present invention allows multipleprocessors or threads to have different completion times for servicingmultiple service requests while maintaining data integrity. While twostorage requests are shown, the present invention is also applicable tothree or more concurrent storage requests accessing the same datasegment. One of skill in the art will appreciate other benefits of thepresent invention and other ways to implement the modules 1202, 1204,1302-1308 of the apparatus 1200, 1300.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

1. A method for coordinating storage requests, comprising: generating a first append storage command in response to a first request to store first data on a storage device; generating a second append storage command in response to a second request to store second data on the storage device, the second data overlapping at least a portion of the first data, the second request received after the first request; and updating an index for the first append storage command and updating the index for the second append storage command, wherein the update for the first append storage command retains a reference to the second data in the index when the index is updated for the first append storage command after the index is updated for the second append storage command.
 2. The method of claim 1, further comprising preventing access to a region of the index pertaining to the second append storage command while updating the index for the second append storage command.
 3. The method of claim 1, wherein updating the index for the second append storage command further comprises: preventing access to a first region of the index pertaining to the second append storage command while updating the index to reference a storage location of the second data on the storage device, such that other regions of the index are accessible while updating the first region of the index; and allowing access to the first region of the index after updating the index for the second append storage command.
 4. The method of claim 1, the index comprising a forward index comprising a plurality of nodes associating logical identifiers with storage locations on the storage device, wherein updating the index for the second append storage command comprises restructuring the forward index to include a node associating a logical identifier of the second request with a storage location of the second data on the storage device.
 5. The method of claim 4, wherein the second data overlapping at least a portion of the first data comprises a first set of logical identifiers that identify a storage location of the second data, the first set of logical identifiers intersecting with a second set of logical identifiers that identify a storage location of the first data, and wherein updating the index comprises restructuring the forward index such that a single node identifies a set of logical identifiers comprising the intersection of the first set of logical identifiers and the second set of logical identifiers.
 6. The method of claim 5, wherein restructuring the forward index comprises splitting one or more nodes of the forward index associating logical identifiers with the portion of the first data overlapping the second data into two or more nodes.
 7. The method of claim 1, the index comprising a forward index comprising a plurality of nodes to associate logical identifiers with storage locations on the storage device, the forward index comprising a node associating a logical identifier with a storage location of the second data on the storage device, and wherein updating the index for the first append storage command comprises updating a node in the forward index to reference a storage location comprising the first data and retaining the node associating the logical identifier with the storage location of the second data.
 8. The method of claim 1, wherein updating the index for the second append storage command comprises updating the index to mark the portion of the first data overlapped by the second data as invalid and updating the index to reference a storage location of the second data on the storage device.
 9. The method of claim 8, wherein marking the portion of the first data overlapped by the second data as invalid comprises updating a reverse index associating storage locations of the storage device with logical identifiers to mark the portion of the first data overlapped by the second data as invalid.
 10. The method of claim 9, wherein the reverse index is organized by storage division, the method further comprising selecting a storage division for recovery based at least in part on a proportion of data marked as invalid in a selected storage division of the reverse index.
 11. The method of claim 1, wherein the second append storage command comprises a command to store the second data sequentially on the storage device to preserve an ordered sequence of storage operations performed on the storage device.
 12. The method of claim 1, wherein the second request comprises a logical identifier of the second data, and wherein the second append storage command comprises a command to store the logical identifier with the second data on the storage device.
 13. The method of claim 1, wherein the first request and the second request originate from different clients.
 14. The method of claim 1, further comprising: updating the index for the first append storage command by way of a first thread; and updating the index for the second append storage command by way of a second thread operating concurrently with the first thread.
 15. The method of claim 1, further comprising: associating the first and the second append storage commands with respective sequence indicators, the sequence indicators indicating that the second append storage command is more recent than the first append storage command; and using the sequence indicators to retain the reference to the second data in the index.
 16. The method of claim 15, wherein the sequence indicators represent an order in which the first and the second requests were received.
 17. The method of claim 15, wherein the first and the second requests comprise the sequence indicators.
 18. The method of claim 15, wherein the sequence indicators represent an order in which the first and the second requests were transmitted.
 19. The method of claim 1, further comprising delaying a request to read data pertaining to the second request until updating the index for the second append storage command.
 20. A method for coordinating storage requests, comprising: generating a first append storage command in response to a first request to store first data on a storage device; generating a second append storage command in response to a second request to store second data on the storage device, the second data pertaining to a logical identifier range that overlaps at least a portion of a logical identifier range of the first data, wherein the second request is more recent than the first request; updating an index associating logical identifiers with storage locations on the storage device for the first append storage command and updating the index for the second append storage command, the updating to retain an association between a storage location of the second data and the overlapping logical identifier range when the index is updated for the second append storage command before being updated for the first append storage command, wherein updating the index comprises invalidating a portion of the first data associated with the overlapping logical address range.
 21. The method of claim 20 further comprising preventing access to a region of the index pertaining to the second append storage command while updating the index for the second append storage command.
 22. The method of claim 20, wherein updating the index for the second append storage command further comprises: preventing access to a first region of the index pertaining to the second append storage command while updating the first region of the index to associate a logical address range of the second data with a storage location of the second data on the storage device, such that other regions of the index are accessible while the first region is updated; and allowing access to the first region after updating the index for the second append storage command.
 23. The method of claim 20, wherein a first thread updates the index for the first append storage command, and a second thread updates the index for the second append storage command.
 24. A computer program product comprising a computer readable storage medium comprising code executable to perform operations for coordinating storage requests, the operations comprising: generating a first append storage command in response to a first request to store first data on a storage device; generating a second append storage command in response to a second request to store second data on the storage device, the second data pertaining to a logical identifier range that overlaps at least a portion of a logical identifier range of the first data, wherein the second request is more recent than the first request; updating an index associating logical identifiers with storage locations on the storage device for the first append storage command and updating the index for the second append storage command, the updating to retain an association between a storage location of the second data on the storage device and the overlapping logical identifier range when the index is updated for the second append storage command before the index is updated for the first append storage command; and preventing access to a region of the index pertaining to the second append storage command while the index is updated for the second append storage command. 